Faculty of Science and Technology¶
Crime Data Anlaysis and Clustering ¶
T. Aljrees, D. Windridge , D. Shi, X. Cheng¶
In [3]:
from IPython.display import HTML
HTML('''<script>
code_show=true;
function code_toggle() {
if (code_show){
$('div.input').hide();
} else {
$('div.input').show();
}
code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()">
<input type="submit" value="Click here to toggle on/off the raw code."></form>''')
Out[3]:
Downlaod Turki Thesis Pdf Tasks and Agenda : ¶
Table of agenda :
- Overview, adn The ideas or concepts (Seriouse 1 )
- Data Insight (Seriouse 2)
- Data Cleaning (Seriouse 3)
- Data Exploration (Seriouse 4 )
- Data Integration (Seriouse 5)
- Feature Transformation (Feature Engineering) (Seriouse 6)
- Feature Selction and model building (Seriouse 7)
- Data Test / Training(Seriouse 8)
- Applying advanced subspace Clustering(Seriouse 9)
In [ ]:
In [7]:
summaryall = pd.read_csv("summaryall.csv",error_bad_lines=False,nrows = 10000)
summaryall
Out[7]:
Note ¶
Data been through many tasks before Clustring and data reduction :
That included preparing data
- Data Claening
- Distribution of Features
- Identfiying missing values in dataset
- Handlled mising data
- visualaisation
- converted categorical and ordinal features into numeric features
- Created dummy features
- Transformed a categorical feature into a set of dummy features, each representing a unique category
- Transformed a categorical feature into a numeric values
- scaling , and Normalisation
This document for :
- Corrleation (scaled and nonscaled )
- Clustering of both data (scaled and nonscaled )
- Drop highly correlated features
- PCA both (scaled and nonscaled ) Feature Extraction
- k-fold , Elbow method to select optimal K
- Checking optimal number of ClustersIn [92]:
# before clustering
IntNoDumKmeans.columns
Out[92]:
In [17]:
scaled_IntNoDumKmean.columns
Out[17]:
Correlation¶
correlation of data¶
In [27]:
#data
#correlation map
f,ax = plt.subplots(figsize=(8,8))
sns.heatmap(data.corr(), annot=True, linewidths=0.5, fmt= '.1f',ax=ax)
plt.show()
correlation of scaled_IntNoDumKmean¶
In [25]:
#correlation map
#note tha ; correlation of scaled_IntNoDumKmeant this is before kmean
f,ax = plt.subplots(figsize=(8,8))
sns.heatmap(scaled_IntNoDumKmean.corr(), annot=True, linewidths=0.5, fmt= '.1f',ax=ax)
plt.show()
correlation of IntNoDum¶
In [125]:
#intdum
#correlation map
f,ax = plt.subplots(figsize=(8,8))
sns.heatmap(IntNoDum.corr(), annot=True, linewidths=0.5, fmt= '.1f',ax=ax)
plt.show()
correlation of dummiesNum¶
In [127]:
dummiesNum.info()
In [126]:
#dummiesNum
#correlation map
f,ax = plt.subplots(figsize=(8,8))
sns.heatmap(dummiesNum.corr(), annot=True, linewidths=0.5, fmt= '.1f',ax=ax)
plt.show()
Plot¶
In [131]:
# plot scaleed correlated values
plt.rcParams['figure.figsize'] = [16, 6]
fig, ax = plt.subplots(nrows=1, ncols=3)
ax=ax.flatten()
cols = ['Offence', 'Street', 'District']
colors=['#415952', '#f35134', '#243AB5', '#243AB5']
j=0
for i in ax:
if j==0:
i.set_ylabel('Beat_Num')
i.scatter(IntNoDum[cols[j]], IntNoDum['Beat_Num'], alpha=0.5, color=colors[j])
i.set_xlabel(cols[j])
i.set_title('Pearson: %s'%IntNoDum.corr().loc[cols[j]]['Beat_Num'].round(2)+' Spearman: %s'%scaled_IntNoDum.corr(method='spearman').loc[cols[j]]['Beat_Num'].round(2))
j+=1
plt.show()
Kmeans¶
Kmeans datset scaled_IntNoDumKmean¶
In [67]:
def doKmeans(scaled_IntNoDumKmean, nclust=5):
model = KMeans(nclust)
model.fit(scaled_IntNoDumKmean)
clust_labels = model.predict(scaled_IntNoDumKmean)
cent = model.cluster_centers_
return (clust_labels, cent)
clust_labels, cent = doKmeans(scaled_IntNoDumKmean, 2)
kmeans = pd.DataFrame(clust_labels)
scaled_IntNoDumKmean.insert((scaled_IntNoDumKmean.shape[1]),'kmeans',kmeans)
In [72]:
fig = plt.figure()
ax = fig.add_subplot(111)
scatter = ax.scatter(scaled_IntNoDumKmean['District'],scaled_IntNoDumKmean['Beat_Num'],
c=kmeans[0],s=25)
ax.set_title('K-Means Clustering')
ax.set_xlabel('District per Beat_Num')
ax.set_ylabel('Beat_Num')
plt.colorbar(scatter)
Out[72]:
In [73]:
fig = plt.figure()
ax = fig.add_subplot(111)
scatter = ax.scatter(scaled_IntNoDumKmean['Street'],scaled_IntNoDumKmean['Beat_Num'],
c=kmeans[0],s=25)
ax.set_title('K-Means Clustering')
ax.set_xlabel('Street per Beat_Num')
ax.set_ylabel('Street')
plt.colorbar(scatter)
Out[73]:
In [74]:
#Offence
fig = plt.figure()
ax = fig.add_subplot(111)
scatter = ax.scatter(scaled_IntNoDumKmean['Offence'],scaled_IntNoDumKmean['Beat_Num'],
c=kmeans[0],s=25)
ax.set_title('K-Means Clustering')
ax.set_xlabel('Offence per Beat_Num')
ax.set_ylabel('Offence')
plt.colorbar(scatter)
Out[74]:
Kmeans datset not Scaled and not dummy¶
In [81]:
def doKmeans(IntNoDum, nclust=5):
model = KMeans(nclust)
model.fit(IntNoDum)
clust_labels = model.predict(IntNoDum)
cent = model.cluster_centers_
return (clust_labels, cent)
clust_labels, cent = doKmeans(IntNoDum, 2)
kmeans = pd.DataFrame(clust_labels)
IntNoDum.insert((IntNoDum.shape[1]),'kmeans',kmeans)
In [118]:
fig = plt.figure()
ax = fig.add_subplot(111)
scatter = ax.scatter(IntNoDum['District'],IntNoDum['Beat_Num'],
c=kmeans[0],s=25)
ax.set_title('K-Means IntNoDum Clustering')
ax.set_xlabel('District per Beat_Num')
ax.set_ylabel('Beat_Num')
plt.colorbar(scatter)
Out[118]:
In [84]:
fig = plt.figure()
ax = fig.add_subplot(111)
scatter = ax.scatter(IntNoDum['Street'],IntNoDum['Beat_Num'],
c=kmeans[0],s=25)
ax.set_title('K-Means IntNoDum Clustering')
ax.set_xlabel('Street per Beat_Num')
ax.set_ylabel('Street')
plt.colorbar(scatter)
Out[84]:
In [85]:
#Offence
fig = plt.figure()
ax = fig.add_subplot(111)
scatter = ax.scatter(IntNoDum['Offence'],IntNoDum['Beat_Num'],
c=kmeans[0],s=25)
ax.set_title('K-Means IntNoDum Clustering')
ax.set_xlabel('Offence per Beat_Num')
ax.set_ylabel('Offence')
plt.colorbar(scatter)
Out[85]:
In [87]:
IntNoDumKmeans = IntNoDum
In [88]:
IntNoDumKmeans.columns
Out[88]:
In [116]:
IntNoDumKmeans[["Street", "District", "Town", "Post_Code", "Offence", "MO_Desc", "Beat_Num", "kmeans"]].head()
Out[116]:
In [114]:
IntNoDumKmeans[["Street", "District", "Town", "Post_Code", "Offence", "MO_Desc", "Beat_Num", "kmeans"]].tail()
Out[114]:
PCA¶
PCA pf scaled_IntNoDumKmean¶
In [121]:
from sklearn.decomposition import PCA
pca = PCA(n_components=19)
pca.fit(scaled_IntNoDumKmean)
Out[121]:
In [122]:
print(pca.explained_variance_ratio_)
plt.figure(figsize=(15,7))
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of IntNoDumKmeans components')
plt.ylabel('cumulative explained variance');
PCA of IntNoDumKmeans¶
In [119]:
from sklearn.decomposition import PCA
pca = PCA(n_components=19)
pca.fit(IntNoDumKmeans)
Out[119]:
In [120]:
print(pca.explained_variance_ratio_)
plt.figure(figsize=(15,7))
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of IntNoDumKmeans components')
plt.ylabel('cumulative explained variance');
PCA pf IntNoDum¶
IntNoDum
In [123]:
#IntNoDum
from sklearn.decomposition import PCA
pca = PCA(n_components=19)
pca.fit(IntNoDum)
Out[123]:
In [124]:
print(pca.explained_variance_ratio_)
plt.figure(figsize=(15,7))
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of IntNoDum components')
plt.ylabel('cumulative explained variance');
PCA of Data¶
In [134]:
data.head()
Out[134]:
In [145]:
data.columns
Out[145]:
In [144]:
data = pd.read_csv("Crime_comma.csv",error_bad_lines=False,nrows = 10000)
In [146]:
#drop
data = scaled_IntNoDumKmean.drop(['Crime_Ref', 'Crime_Num'], axis=1)
In [147]:
## EROR becuse could not convert string to float: '99Y4/463197/19'
## we need to drop string
from sklearn.decomposition import PCA
pca = PCA(n_components=18)
pca.fit(data)
Out[147]:
In [148]:
print(pca.explained_variance_ratio_)
plt.figure(figsize=(15,7))
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('number of IntNoDum components')
plt.ylabel('cumulative explained variance');
Computes the Euclidean distance between two 1-D arrays.
The Euclidean distance between 1-D arrays u and v, is defined as
.. math::
$${||u-v||}_2$$$$\left(\sum{(w_i |(u_i - v_i)|^2)}\right)^{1/2}$$Manhattan distance between two 1-D arrays u and v,which is defined as
.. math::
$$\sum_i {\left| u_i - v_i \right|}.$$
and known inpython as dist.cityblock(AA, BB)
Parameters
u : (N,) array_like Input array.
v : (N,) array_like Input array.
w : (N,) array_like,
optional
The weights for each value in u and v. Default is None, which gives each value a weight of 1.0
Chebyshev -- also chessboard -- distance is best defined as a distance metric
math::
$$ \max_i {|u_i-v_i|}.$$
the Cosine distance Compute distance between 1-D arrays. between u and v, is defined as
math::
$$ 1 - \frac{u \cdot v} {||u||_2 ||v||_2} $$where :math:u \cdot v is the dot product of :math:u and
:math:v.
In [56]:
import scipy.spatial.distance as dist
dist.euclidean(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence)
Out[56]:
In [58]:
print('\nEuclidean distance is =', dist.euclidean(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence))
print()
print('Manhattan distance is =', dist.cityblock(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence))
print()
print('Chebyshev distance is =', dist.chebyshev(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence))
print()
print('Canberra distance is =', dist.canberra(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence))
print()
print('Cosine distance is =', dist.cosine(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence))
In [60]:
#Chebyshev -- also chessboard -- distance is best defined as a distance metric
dist.cityblock(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence)
Out[60]:
In [62]:
dist.chebyshev(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence)
Out[62]:
In [63]:
dist.cosine(scaled_IntNoDumKmean.District, scaled_IntNoDumKmean.Offence)
Out[63]:
$$a = hav(Δφ)+cos(φ1)⋅cos(φ2)⋅hav(Δλ) $$
In [ ]: